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Hankel Determinants for Some Common Lattice Paths 

Robert A. Sulanke and Guoce Xin 

Abstract. For a single value of £, let f{n,£) denote the number of lattice paths that use the steps (1, 1), 
(1, —1), and {£, 0), that run from (0, 0) to (n, 0), and that never run below the horizontal axis. Equivalently, 
/(n, £) satisfies the quadratic functional equation F{x) = '}2n>0 ^)^'" = 1 + x^F{x) + x'^F(x)'^. Let H„ 
denote the n by n Hankel matrix, defined so that [H„]i^j = f{i + j — 2, £). Here we investigate the values of 
such determinants where £ = 0,1, 2, 3. For £ = 0,1,2 we are able to employ the Gessel-Viennot-Lindstrom 
method. For the case £ = 3, the sequence of determinants forms a sequence of period 14, namely, 

(det(H„))„>i = (1, 1, 0, 0, -1, -1, -1, -1, -1, 0, 0, 1, 1, 1, 1, 1, 0, 0, -1, -1, -1, . . .) 

For this case we are able to use the continued fractions method recently introduced by Gessel and Xin. We 
also apply this technique to evaluate Hankel determinants for other generating functions satisfying a certain 
type of quadratic functional equation. 

Resume. Pour une seule valeur de £, soit /(n, £) le nombre des chemins treillis que utilise les pas (1,1), (1,-1), 
et {£, 0), vient de (0, 0) a (n, 0), et que ne vient jamais dessous I'axis horizontale. Equivalentement, le f{n, £) 
satisfie I'equation fonctionnelle quadratique F{x) = X],i>o /('^i'^)^" = 1 + x^F{x) + x^F{x)^. Soit H„ le n 
par n matrice de Hankel, definit pour que [Hn]ij = f{i + j — 2, £). Nous examinons de tels determinants ou 
£ = 0,1, 2, 3. Pour £ = 0,1,2 nous pouvons employer la methode de Gessel-Viennot-Lindstrom. Pour le cas 
£ = 3, Is sequence de determinants forme une sequence de periode 14, a savoir 

{det(H„))„>i = (1, 1, 0, 0, -1, -1, -1, -1, -1, 0, 0, 1, 1, 1, 1, 1, 0, 0, -1, -1, -1, . . .) 

Pour ce cas que nous pouvons utiliser la methode de fractions continuee recemment introduit par Gessel et 
Xin. Nous appliquons aussi cette technique pour evaluer les determinants de Hankel pour I'autres fonctions 
generatrices quie satisfait un certain type d'equation fonctionnelle qudratique. 



1. Introduction 

We will consider lattice paths that use the following three steps: U = (1,1), the up diagonal step; 
H — {£,0), the horizontal step of length £, where ^ is a single nonnegative integer; and D — (1, —1), the 
down diagonal step. Further, each H step will be weighted by t, and the others by 1. The weight of a path 
is the product of the weights of its steps. The weight of a path set is the sum of the weights of its paths. 

Let f{n, t, £) denote the weight of the path set of paths running from (0, 0) to (n, 0) that never run below 
the X-axis. When t = I, weight becomes cardinality. For example, 

- /(n,0, 0), equivalently f{n,0,£), is the weight of a set of Dyck paths, counted by the aerated 
Catalan numbers: 

(/(0, 0, 0), /(1, 0, 0), /(2, 0, 0), . . .) = (1, 0, 1, 0, 2, 0, 5, 0, 14, 0, 42, 0, 132, 0, 429, . . .). 

- f{n, t, 1) is the weight of a set of Motzkin paths, counted by the Motzkin numbers: 
(/(0, 1, 1), /(1, 1, 1), /(2, 1, 1), . . .) = (1, 1, 2, 4, 9, 21, 51, 127, 323, 835 . . .). 

- f{n,t,2) is the weight of a set of large Schroder paths, counted by the aerated large Schroder 
numbers: 

(/(0, 1, 2), /(1, 1, 2), /(2, 1,2),...) = (1, 0, 2, 0, 6, 0, 22, 0, 90, 0, 394, 0, ). 

- (/(0, 1, 3), /(1, 1, 3), /(2, 1,3),...) = (1, 0, 1, 1, 2, 3, 6, 10, 20, 36, 72, 136, 273, 532, . . .) 
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Previously, Pergola, et al [S] and Sulanke have considered such generalized Motzkin paths for various 
values of £ and have given additional references. Letting 

n>0 

denote the generating function for f{n,t,£), we find by a common combinatorial decomposition that F{x) 
satisfies the functional equation 

F{x) = 1 + tx'''F{x) + x^F{xf. 

Any sequence A — (ao, ai, a2 . . .) defines a sequence of Hankel matrices, iJi, -ff2, ^^3 • ■ • , where iJ„ is an 
n by n matrix with entries {Hn)ij = ai+j-2- For instance, the sequence (/(rt, l,3))„>o yields 

"1011' 

n 01 



Hi = [1] , H2 = 



1 



Hi = 



1 1 

1 1 

1 1 2 



Hi = 



112 
112 3 
12 3 6 



Our interest is to consider, for nonnegative integer £, the corresponding sequence of determinants det(_ff„) 
where each matrix _ff„ has entries 

(i/„),,, =/(j+j-2,i,£). 
The following propositions constitute our main results: 

Proposition 1.1. For n > 0, £ — \, and arbitrary t (including t = 0, yielding the Dyck path case) 

det{Hn) - 1. 

Proposition 1.2. For n > 0, £ = 2, and arbitrary t (including t — 0, yielding the Dyck path case), 



det{H„) = 



(1 
(1 



t)(n-l)("+l)/4 



if n is even 
if n is odd 



Proposition 1.3. For < = 1 and £ ^ 3, 

\14 



(dct(i/„)Xf>i = (1, 1, 0, 0, -1, -1, -1, -1, -1, 0, 0, 1, 1, 1). 

Moreover, if m,n > with n m — mod 14 then det(Hm) = det(i7„). 

In Section 13 using the well-known combinatorial method of Gessel-Viennot-Lindstrom |3] [5] |13| . we 
will prove Propositions 1 1 . II and 11.21 Our proof of Propositions lTTTl is essentially that of Viennot |13j who also 
used the method to calculate various other Hankel determinants relating to Motzkin paths. Aigner ^ also 
studied such determinants. We note that earlier Shapiro 10 demonstrated that the Hankel determinants 
for the usual Catalan numbers is 1. For the large Schroder numbers (r(n))„>o — 1, 2, 6, 22, 90, 394, . . . whose 
generating function satisfies 

R{x) = r{k)x'' = 1 + xR{x) + xR{x)^, 

k>0 

we show that the n-order Hankel determinant 

is 2»("-i)/2^ as stated in Proposition 12. II 
We remark that the problem of evaluating Hankel determinants corresponding to a generating function 
has received significant attention as considered by Wall |14| . One of the basic tools for such evaluation is 
the method of continued fractions, either by J- fractions in Krattenthaler |S] or Wall |14j or by S"- fractions in 
Jones and Thron [3 Theorem 7.2]. However, both of these methods need the condition that the determinant 
can never be zero, a condition not always present in our study. Recently, Brualdi and Kirkland 4 used the 
J-fraction expansion to calculate Hankel determinants for various sequences related to the Schroder numbers. 
A slight modification of their proof of W, Lemma 4.7] proves our Proposition 12 . II for t = 1. 

In Section Owe establish the periodicity of 14 for the case £ = 3 of Proposition 11.31 by the continued 
fraction method recently developed by Gessel and Xin 6 . In the final section, we review their technique 
more generally: it yields a transformation for generating functions, satisfying a certain quadratic functional 
equation, that also transforms the associated Hankel determinants in a simple manner. We apply this trans- 
formation to evaluate the Hankel determinants for the cases ^ = 1, 2 (again) and for other path enumeration 
sequences related to £ = 3. 
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Figure 1: Some of the 4-tuples of paths for £ = 1 and for I-T-CONFIG with [(0, 0), (-1, 0), (-2, 0), (-3, 0)] and 
[(0, 0), (1, 0), (2, 0), (3, 0)]. In each of these 4-tuples there is a point path (a path of zero length) at (0, 0). The 
first 4-tuple is the only nonintersecting 4-tuple for this case. The second and third 4-tuples are intersecting 
only at the point (0, 1). The second 4-tuple corresponds to the permutation 1243 having sign of -1, while the 
third corresponds to the permutation 1342 having sign of 1. These two 4-tuples cancel one another under 
the Gessel-Viennot-Lindstrom method. 



2. Employing the Gessel-Viennot-Lindstrom method 

Assuming a rudimentary knowledge of the Gessel-Viennot-Lindstrom method, we reformulate it to our 
needs. All lattice paths use the three steps as previously defined. Given an n-tuple of lattice paths on the 
Z X Z plane, we say that it is nonintersecting if no steps from different paths share a common end point. 
Thus an nonintersecting n-tuple may have paths crossing or touching at points other than a common step 
end point. 

Let [{xi,yi), {x2,y2), • • ■ , [xn^Vn)] and [{x{,y[), ix2,y'2), {Xn,y'n)] denote two lists of distinct lattice 
points such that 

Xk+i <Xk<0 and < yk < yk+i 

and 

^<x'k< 4+1 and Q<y'k< y'k+i- 

We will refer to such a pair of lists as an "i-t-CONFIG" of order n as their points will be the initial and 
terminal points for each rt-tuplc of paths being considered. 

Let Pij denote the set of all paths running from {xi, yi) to (x^ , y'^) that never run below the x-axis, with 
\Pij I denoting the sum of the weights of its paths. Let Sn denote the set of permutations on {1, 2, 3, . . . , 7i}. 
For any permutation a e S'„, let denote the set of all n-tuples of paths (pi,p2, • ■ ■ ,Pn), where pi e Pi,a{i) 
for 1 < i < n. The signed weight of (pi,p2, ■ ■ ■ ,Pn) S P^ is defined to be sgn(cr) times the product of the 
weights of the n paths. See Figures ^ and |21 

For our purpose the Gessel-Viennot-Lindstrom method is formulated in a form similar to that in Vien- 
not's notes |13j : 

Lemma 2.1. Given an i-t-CONFIG of order n, the sum of the signed weights of the nonintersecting 
n-tuples in Ua-eS„Pa equal to det( {\Pij\)i<ij<n )■ 



Proof of Proposition II .11 (A similar proof appears in [T3].) By Lemma ITTl detfg^'l is equal 
to the sum of the signed weights of the nonintersecting n-tuples in Ua-eSnPo- for the i-t-CONFIG where 
{xi,yi) = {-i + 1,0) and (a;-,y-) = {i - 1,0), for 1 < i < n. Thus, for this i-t-CONFIG, we seek the 
nonintersecting n-tuples. First, the 1-tuple Pi.i contains just the point path beginning and ending at (0, 0). 
Next, any nonintersecting path from (— i + 1,0), for 1 < i < n, must begin with an U step, while any 
nonintersecting path to (j — 1,0), for 1 < j < n, must end with an D step. Repeating this analysis at 
each integer-ordinate level k, shows the nonintersecting path from {—i + 1,0), 1 < i < k, is forced to be a 
sequence of U steps followed by a sequence of D steps; moreover, it shows that any nonintersecting path 
from {—i + 1,0) to (j — 1,0), k < i,j, must start with k U steps and end with k D steps. Inductively, each 
nonintersecting path is a sequence of U steps followed by a sequence of D steps. The n-tuple of such paths 
is the only nonintersecting n-tuple of Ua-eSnPa, and it has weight equal 1. □ 

We will use the following in proving Proposition ^21 
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Lemma 2.2. For the lattice paths that use the steps U, H — (2,0) , and D, that never run below the 
X-axis, and that have the I-T-CONFIG, 

(x„ y,) = {-2i + 2, 0) and {x'^,y[) = {2i - 2, 0) 

for 1 < i < n, the sum of the signed weights of the nonintersecting n-tuples in UcreS^Pcr equals + 

Proof. For {pi,p2, ■ ■ • ,Pn) e {Pi,a{i},P2,a{2), ■ ■ ■,Pn,G{n)), supposc that (pi,_p2, • • ■ ,Pn) is a noninter- 
secting n-tuple of paths for some permutation a. Since the points in the i-t-CONFIG are spaced two units 
apart, the horizontal distance at any integer ordinate between any two paths of (Pi,(t(i), ^2,(7(2)1 • ■ ■ j Pn,a{n)) 
must be even. It follows inductively that, for 1 < i < n, any path of the path set Pi^a(i) must begin with a 
sequence of i — 1 ?7-steps and finish with a sequence of a{i) — 1 -D-steps. Thus, computing the weight of the 
nonintersecting n-tuples is equivalent to computing the weight of the nonintersecting n-tuples for the new 
("V" shaped) initial-terminal configuration, denoted by i-t-CONFIG-new, defined by 

(a^j, Vi) = (-i + 1, i - 1) and {x'^, y-) = (i - 1, « - 1) 

for 1 < i < n. 

Before continuing, we notice, for example when t = 1 and n — 4, that the matrix M (0) define by 
(M(0)ij)i<ij<4 = {\Pi,j\)i<i.3<i for i-t-CONFIG-new is an array of Delannoy numbers. (See [H].) 
When t = 0, M(0) is the initial array from Pascal's triangle. In the following array for t = 1, the entries 
count the ways a chess king can move from the north-west corner if it uses only east, south, or south-east 
steps. Momentarily we will see the role of the argument in M(0). 



M(0) = 



1111 
13 5 7 
1 5 13 25 
1 7 25 63 



Now for arbitrary t and n, let M(0) be the n by n matrix defined recursively by 

Af(0),,, = M(0),_i,, +tAf(0),_i,,_i -I- Af(0),j_i 

for 1 < z and 1 < j with Af(0)ij = 1 and M(0),,i = 1 for 1 < i and 1 < j. By Lemma lO M (Q) = 
{\Pij\)i<i,j<n for i-t-CONFIGNew. Thus det(M(0)) is equal to the weight of the nonintersecting n-tuples 
for I-T-CONFIG. The proof is completed once we show 

det(M(0)) = (1 + i)"^""'^/'. 

Given Af (0), we recursively define a sequence of n by n matrices 

A^(0), A//(l), A//(2), ...,A/(n-l) 

where, for 1 < < n — 1, 

Af(fc-l)jj for 1 < i < fc 



-y ^^^^ _ _ j^j^^ _ ^^^^^^^ iork + \<i<n 
With CLAiM(fc) denoting the claim that 

Af(fc),j ^ Af(fc),_ij +tAf(fc),_ij_i + Af(fc),,j_i for i,j > k, 
M{k),^, = {l+ty-^ ioTi<k, 
^{^)i,3 — for i> i and j < k, 
M{k),^k+i = {l+tf iov i>k + l, 

one can establish CLAiM(fc) for 1 < fc < n — 1 by induction. Since Af(n — 1) is upper triangular, we observe 
that 

det(Af(n-l)) = (l + i)"("-i)/^ 
By the type of row operations used to obtain the sequence Af (0), M{1),M (2), . . . , A/(n — 1), their determi- 
nants are equal. □ 

Since, by the i-t-CONFIG of Lemma {H)i,j = \Pi,j\ counts the large Schroder paths from (0,0) to 
(2i -I- 2j — 2,0), immediately we have the the following corollary for the Hankel determinants of the weighted 
non-aerated Schroder numbers: 
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Proposition 2.1. Let fn denote the weight of the path set of paths from (0,0) to (2n,0) which never 
run beneath the x-axis and where H = (2, 0) is weighted by t. Equivalently, let /„ satisfy 

F{x) = ^ /„a;" = 1 + txF{x) + xF{xf. 
n>a 

Then the determinant of the n-th order Hankel matrix equals (1 + t)"("^i)/2. 
As a second corollary to Lemma [2. 21 we have 

Lemma 2.3. For the lattice paths that use the steps U, H = (2,0), and D, that never run below the 
X-axis, and that have the i-t-CONFIG with 

{x^.Vi) = {-2i + 1,0) and {x[,y[) = {2i - 1,0) 

for 1 < i < n, the sum of the signed weights for the nonintersecting n-tuples in Uo-gS„-P(t is (1 + t)"("+i)/^. 

Proof of Lemma 12.31 We first translate all paths upwards one unit and then prepend a f7-step and 
append a I?-step to every path. Next we add the point path at (0,0). The sum of the signed weights of the 
nonintersecting n-tuples in the original configuration equals that of the nonintersecting n + 1-tuples in this 
new configuration, which in turn is given by Lemma 12.21 □ 

Proof of Proposition fTTSl Suppose that n is even; the proof when n is odd is similar. Here the 
Hankel matrix {\Pi.j\)i<ij<n corresponds to the i-t-CONFIG with 

{xi,yi) = {—i + 1, 0) and {x[,yl) = (i — 1, 0) for 1 < i < n. 

Since i ^ 2, no endpoint of a step on a path that originates from an oddly indexed initial point (i.e., a point 
(— i + 1,0) for odd i) will intersect an endpoint of a step on a path that originates from an evenly indexed 
initial point. Moreover, for any permutation a corresponding to a nonintersecting n-tuple, a{i) — i must be 
even for each z, and hence sgn(cr) = 1. Thus the weight of the nonintersecting n-tuples is the product of 
the weight of those originating from oddly indexed initial points times the weight of those originating from 
evenly indexed initial points. 

Hence, with to = n/2, let i-t-CONFIgA have 

(xi,yC) = {-2i + 2, 0) and (x-, y[) = {2i - 2, 0) for 1 < i < to, 

and let i-t-CONFIGB have 

(xi,yC) = {-2i + 1,0) and (x-,?/-) = {2i - 1,0) for 1 < i < to. 

Applying Lemmas 12.21 and 12.31 to these configurations yields the weight of nonintersecting n-tuples of the 
original configuration as 

(1 + t)'^(™-l)/2(l + ^yn(m+\)l2 = (X i)™' = (1 + i)"V4^ 

□ 



Next we consider Hankel determinants for sequences of path weights that ignore the initial term. For 
the sequence f {l,t,i), f {2, t, £),... , we will let denote the matrix where the entries satisfy {H^)ij = 
/(i + j - 1, i, £). See Figure|21 

Proposition 2.2. Fori = 1 (Motzkin case again), the sequence of determinants satisfies the recurrence 

det(i/i) - t det(i7i_i) - det(i/i_2) 

subject to det{Hl) = t and det{H^) = (i - l)(i + 1). 

Proof. Aigner [J considered the case for t = 1. For arbitrary t, our proof considers how the partic- 
ular paths must look in the nonintersecting case. Observe that det{H^) is the sum of the weights of the 
nonintersecting n-tuples for the i-T-CONFiG(n) taken as 

[(0, 0), (-1, 0), . . . , (n - 1, 0)] and [(1, 0), (2, 0), . . . , (n, 0)]. 

Each of these nonintersecting n-tuples belongs to one of two types: (1) those containing the path from (0, 0) 
to (1, 0) with all other paths forced to begin with U, end with D, and have ordinate at least one elsewhere; 
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Figure 2: Three of the 4-tuples of paths for ^ = 1 and for I-t-CONFIG with [(0,0), (-1,0), (-2,0), (-3,0)] 
and [(1, 0), (2, 0), (3, 0), (4, 0)]. The first and second 4-tuples are both nonintersecting. The first has a signed 
weight of t"^ while the second has a signed weight of — t^. The third is intersecting only at the point (0, 1). 



(2) those containing the path U D from (0, 0) to (2, 0) and the path UD (—1,0) to (1,0) with all other paths 
forced to begin with UU, end with DD, and have ordinate at least two elsewhere. The set of the first type 
has a total weight t times the sum of the weights of the nonintersecting (n — l)-tuples on the i-T-CONFiG(ri-l), 
which is t det(i?,j_]^). Since each n-tuple of the second type has the defined crossing of the path from (0, 0) 
with that from (—1, 0), the set has total weight is the sign of the corresponding permutation times the sum 
of the weights of the nonintersecting (n — 2)-tuples on the i-T-CONFiG(n-2), which is — det{H^_2)- □ 

For £ — 2, we will indicate how Lemma [2 .31 proves 

Proposition 2.3. For n > 0, £ = 2, and arbitrary t, the sequence of determinants satisfies 
, , 1, _ J if n is odd 

det(iJ„) = j (_^yV2(l + ^)n(„+2)/4 ^ g^g^ 

Proof. Here the Hankel matrix can correspond to i-t-CONFIG with 

{xi, yi) = {—i + 1, 0) and (a;,-, j/-) ~ (i, 0) for 1 < i < n. 

Since £ = 2, if there is a path from {xi,yi) to {xj,yj), then i — j is odd. It follows that, if n is odd, there 
can be no n-tuples of paths for the configuration. If n is even and m = n/2, the sign of any permutation 
for an nonintersecting n-tuples can be shown to be (—1)™. Thus the weight of the nonintersecting n-tuples 
is (—1)™ times the weight of those originating from oddly indexed initial points times the weight of those 
originating from evenly indexed initial points. The proof is completed by applving l2.3l to i-t-CONFIgA with 

ixz,yz) = {-2i + 2,0) and (a;^,?/-) = (2i,0) for 1 < i < m, 

and to I-t-CONFIgB with 

ix^,y^) ^ {~2i + 1,0) and {x[,y[) = {2i - 1,0) for 1 < i < m. 

□ 

3. Periodicity fourteen and continued fractions 

Here we will repeated apply the "continued fractions method" recently developed by Gessel and Xin "H" to 
determine the periodicity of the sequence of Hankel determinants for £ = Z and t — 1. This method, presented 
more formally in the next section, transforms both generating functions and corresponding determinants. In 
this section we will concentrate on the specific generating function F{x) satisfying 

F{x) = 1 -I- x^F{x) + x^F{xf. 

From this functional equation, or from the related recurrence for its coefficients, there appears to be no clue 
why the associated sequence of Hankel determinants should have a period of 14. 

For an arbitrary generating function D{x,y) — ^^j^^dijx'^y^ , let [D{x,y)]n denote the n by n deter- 
minant det(((iij)o<ij<ri-i). For any A{x) — X]n>o '^nx", define the Hankel matrix for A of order n, n > 1, 
by H„{A) — (ai+j-2)i<i j"<n- It is straight forward to show that the Hankel determinant det(i/„(j4)) can be 
expressed as 

det(i7„(A)) = \ -^(-)~y^(yy 

x-y 
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We will use an easily-proven "product rule" of for transforming the generating functions: If u{x) is 
a formal power series with u(0) — 1, then 

[u{x)D{x,y)]n = [D{x,y)]n = [u{y)D(x,y)]n. 
We will make five transformations showing, for n > 8, 





" 


1 




|^diag|^[l],[l], 





1 






1 


-2 





where the right side is the determinant of a block-diagonal matrix consisting of six blocks along the diagonal, 
four of which are 1 by 1 identity matrices, and having entry elsewhere. It then follows that det{Hn{F)) — 
— det{Hn-7{F)). This implies that the period for det{Hn{F)) is 14, and Proposition II. 31 will be proved. 

We start with Fo{x) — F{x), and define Fi{x) from Fi^i{x) according to a transformation where each 
Hankel determinant for Fi{x) are derived from one for Fi^i{x) with the aid of the product rule, which is not 
always mentioned. In the following, Fi{x) will always satisfy a quadratic functional equation 

a{x)F,{xf + b{x)F,{x) + c{x) = 0, 

which is equivalent to the continued fraction 

-~c{x) 



In particular, for i — 



F,{x) 
Fo{x) 



b{x) + a{x)F^{x) ' 
1 



1 — a;-^ — x'^Fo{x) 

Transformation 1: Using this continued fraction of Fq, substitution, and simplification we obtain 



detiHniF)) 



xFo{x) - yFo(y) 



x-y 



-xy'^Foiy) + yx'^Fo{x) + {x-y) {yx^ + xy"^ + l) 



(1 - .t3 - x^Fo{x)){l - y3 - y^Fo{y))ix - y) 

Multiplying by (1 — a;^ — x"^ Fq{x)){1 — y^ — y^-Fo(y))i which will not affect the value of the determinant by 
the product rule, we can write the determinant as 

xFi{x) - yFi{y)' 



1 -I- xy- 



x-y 



where 

(3.1) Fi{x)=Fo{x)+x. 

The associated matrix is block-diagonal with two blocks: the matrix [1] and the Hankel matrix for Fi{x). 
Certainly, 

det(i/„(^b)) =det(i/„_i(Fi)). 
From (|3.1|l and the functional equation for Fq[x), we obtain the functional equation 

l + x 



Fi{x) = 



1 -I- a;3 - x'^Fi{x) ' 



Transformation 2: Using this continued fraction for Fi, substituting in 



xFi{x) ~ yFi{y) 
x~y 



and multi- 



plying by {1 + x^ - x'^Fi{x)){l +y^ ~ y^Fi{y)) yields 

'xFi{x) - yFiiy)] \ -xy^ {x + 1) Fi{y) + yx'^ {y + 1) Fi{x) - {y + 1) {x + 1) {xy - 1) {x ~ y) 



x-y 



x-y 



Upon multiplying by (1 -f- a;) ^(1 + y) ^, the determinant is equal to 



1 + xy 



xF2{x) - yF2{y) 
x-y 



where 
(3.2) 



F2{x)^Fi{x)/{l + x)-l. 
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The associated matrix being block diagonal shows 

det(i?„_i(Fi)) = det(iJ„_2(i^2)). 
From (|3.2|l and the functional equation for Fi (x) , we obtain 



1 - 2a;2 - a;3 - {x^ + x^) F2{x) ' 
Transformation 3: Substituting for F2 with the above fraction, simplifying, and multiplying by (1 
x){\ — X — x'^ — x^ F2{x)){\ y){l — y — y'^ — y'^F2{y)) shows that the determinant 



xF2{x) - yF2[y) 



x-y 



equals 



y'^x^ {y + 1) F2{y) - x^y^ {x + 1)^2(0;) - {x - y) {2y'^x^ - x^ - xy - y^) 



which can be rewritten as 



x-y 



2 , ,2 o 2 2 , 3 s^^si^) - yPaiy) 

X + xy + y — 2x y +x y - 



1 

1 

1 -2 



followed by 



x-y 

where F3{x) is indeed a power series satisfying 
(3.3) F3{x)^ix + l)F2ix)/x^. 

This time the corresponding matrix is a block-diagonal matrix with the block 
the Hankel matrix for i^3(x). Hence 

det(iJ„_2(F2)) = -det(i/„_5(i^3)). 
From (|3.3|l and the functional equation for F2{x), we obtain 

p ( \ ^ 1 + ^ 

^^""^ l-2x^ ~x^ -x^Fsix)' 

Transformation 4: Substituting for F3 with the fraction, simplifying, and multiplying by (1 — 2x^ 



— 2:^^3(2;)) ((1 — 2j/^ — y^ — y'^Fsiy)) the determinant 



xFsix) - yFs{y) 



x-y 



equals 



- V + 1) Fz{y) + yx^ {y + 1) F:i{x) + [y + I) {x + 1) {xy + I) [x - y) 



x-y 



By multiplying the generating function by (1 + x) ^(1 + y) ^, this determinant becomes 

xF4^{x) - yFi{y) 



1 + xy- 



x-y 



Fi{x) =^l + x'^F3{x)/{l + x). 



where 
(3.4) 
Therefore, 

det(i/„_5(i^3)) = det(i/„_6(^^4)). 
From (|3.4|l and the functional equation for F^lx), we obtain 

1 



Fi{x) = 



1 + 2:3 - (x3 + p^(^^^ ■ 

Transformation 5: Substituting for F4 with the above fraction, simplifying, and multiplying by (1 



2;^i^4(2;))(l - y'^Fi^{y)) the determinant 



2;F4(2:) -yFiiy) 



x-y 



equals 



-xy"^ {y + 1) FA{y) + x'^y {x + 1) ^4(2;) - {x-y) {yx'^ + xy'^ - l) 

x-y 

where ^5(2;) = (1 + 2;)F4(x) - 2;. Hence, detiHn^eiFi)) = det(ff„_7(F5)). 



1 + xy 



xFr, (x) - yF5 (y) 
x-y 
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Finally, it is routinely checked that ^5(0;) = Fo{x). 

4. The quadratic transformation for Hankel determinants 

One can use the method introduced in the previous section to evaluate the Hankel determinants for 
generating functions satisfying a certain type of quadratic functional equation. The generating functions 
F{x) in this section are the unique solution of a quadratic functional equation satisfying 

U(X) + x'^v[x)}:< [x] 

where u{x) and v{x) are rational power series with nonzero constants, d is a nonnegative integer, and fc is a 
positive integer. Note that if /c = 0, F{x) is not unique. Our task now is to derive a transformation T so 
that det{i?„(F)) = a dei{Hn-d-i{T [F))) for some value a and nonnegative integer d. In addition to Hankel 
matrices for the power series A = Xln>o^^'^'' ^^^^ consider shifted Hankel matrices: H^^{A) denotes the 
matrix {ai+j+k-2)i<i,j<n- Shifted matrices have appeared in Proposition 12 . 21 and 12 . 31 
The first proposition is elementary: 

Proposition 4.1. If F satisfies (|4.1(l . then G = u{0)F satisfies 
det(i?„(G')) = u(0)"det(i/„(i^)), anrf G{x) = 



u{0)-^u{x) + x''u{0)-'^v{x)G{x) ■ 

Proposition 4.2. Suppose F satisfies (|4.1|l with u(0) = 1. We separate u{x) uniquely as u{x) = 
ul{x) + x'^^'^uh{x), where ul{x) is a polynomial of degree at most d + 1 and uh{x) is a power series. 

(i) If k = 1, then there is a unique G such that 

_ ~v{x) - xul(x)uh{x) 
^"^^ ~ ul{x) - x'i+^UH{x) - x<i+^G{x) ' 

Moreover, 

G{x) = -xuh{x) - x~'^v{x)F{x) 
and a shifted matrices appears with 

dei{Hl_,_,{G{x))) = {-lY^'+'y' det{H^{F{x))). 

(ii) If k > 2, then there is a unique G such that 

■^''^'^v{x) - ul{x)uh{x) 



G{x) = 



Moreover, 
and 



ul{x) - x'i+^UH{x) - x'^+^G{x) ' 
G{x) = -uh{x) - x''~'^'^v{x)F{x) 



det{Hn-d-i{Gix))) = (-l)'^('^+i)/2det(i7„(F(a;))). 
Proof. We prove only part (ii) as part (i) is similar. The generating function for Hn{F) is given by 
xF{x) - yF{y) 1 ( x'^+^ 



x — y x — y \u{x) + x''v{x)F{x) u{y) + y'^v{y)F{y) ^ 

_ -/+1-u(.t) - y'^+'^x''v{x)F{x) + x'^+'^u{y) + x'^+^y''v{y)F{y) 
{u{x) + x''v{x)F{x)) {u{y) + y''v{y)F{y)) {x - y) 

We can multiply by {u{x) + x''v{x)F{x)) and by {u{y)+y'^v{y)F{y)) without changing the above determinant 
by the product rule. Next we observe that x"^ divides F{x), and write u{x) = ul{x) + x'^+'^u/f (x) as in the 
proposition. The resulting generating function can be written as 

-yd+^ULjx) + x^+^ULjy) ^ ~^Mx) ~ x''-^-^vix)Fix)) + yjuniy) + (j/)) 
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We now set G{x) — —uh{x) — x v{x)F{x), which can be straightforwardly shown to agree with 



the defining functional equations. Suppose that ul{x) = 1 + aix + ■ ■ ■ Ud+ix"^^^ , then 
equal to the determinant of the block-diagonal matrix 

// ••• 1 \ \ 



xF{ x) - yF {y) 
x-y 



diag 



1 
V 1 ai 



ad-i 

CLd J 



,H{G{x)) 



The determinant of the first block is easily seen to be (-l)'^('^+i)/2. 



□ 



Given these propositions and that H^{A) — H{x ^{A{x) — A{Q))) for any series A, we can now define 
our transformation T{F): For F satisfying (14.1(1 . 

• if u{0) 1, then T{F) — G, as given in Proposition \4. 1\ 

if u{0) — 1 and k — 1, then T{F) = x~^{G{x) — G(0)), with G given in Proposition \4 ■ ^ i ) ■ 
if u{0) = 1 and k>2, then T{F) = G, as given in Proposition \4-.^ ii). 

Moreover, the relation between det{Hn{F)) and det(_ff„(T(F))) is given in Propositions \4 l] and \4.^ 

Example 1: Other proofs of Propositions fTTI and 12 .21 For Motzkin paths with arbitrary the 
generating function F{x) satisfies 

F(x) = - 

^ ' l-tx-x^F{x)' 

Applying Proposition 14 . 21 so Fi — T{F) gives 

det(H„_i(i^i)) = det{Hn{F)) where Fi{x) ^ 



l-tx-x'^Fi{x)' 

Hence, F{x) — Fi{x), and consequently det(7J„(F(a;))) = 1 for all n. 

Whereas the Gessel-Viennot-Lindstrom method leads to a proof in the shifted case for arbitrary t, as in 
Proposition 12. 21 we have been able to use the continued fractions technique only for t = 1 and t = 2. 

For t = 1 we will show that (det(7?^(F)))„>i = (1,0,-1,-1,0,1,1,...), continuing with period 6. Let 
Gi{x) = {F{x) - l)/x, so that det{Hl{F)) = det(i/„(Gi)). Let G2 = r(Gi) and G3 = T(G2), both under 
Proposition 14. 2r ii) . Since 

'^^^^^ " 1 - X - 2x^ - x^Giix) 
with d = 0, k = 3, u{x) = ul{x) = 1 — 2x, uh = 0, and v{x) = —(1 + x)^^, we find that 

X 



G2ix) 



l~x~2x^ ~x^{l + x)G2{x) 



with d — 1, k — 2, u{x) = ul{x) = 1 — x — 2x^, uh = 0, and v{x) — —(1 + x). Applying Proposition 14. 2r iil 
shows 

G^ix) = -x-\-{l + x))G2{x) 

= -x-\-{l + x)){-x{-{l + x)-^))G,{x) 
= Gi{x) 

and det(if„_3(G3)) = — det(iy„_i(G2)) = — det(iJ„(Gi)), which yields the periodicity of the sequence of 
determinants. 

For t = 2 we will show that det{H^{F)) = n + 1 for n > 1. Define, Gi to satisfy, 



Gi(x) = 



l-2x-2a;2-j;3Gi(x)' 
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One can easily see that Gi(x) = {F{x) - l)/x with Gi(0) = ui{0)-^ = det(iJi(Gi)) = det(iJii(i^)) = 2. For 
n > 2, define, G„ to satisfy, 

Q (n - 1)^(^2 + n + a;) 

(n^ — n){n? — 2ii?x — 2x'^) — n^in^ — n + x")x'^Gn(x) 

By induction one can show that G„ = T o T(G„_i) (under Prop. ITTl then under Prop. and that 

Gn(0) = M„(0)^^ — (n — l)(n + l)/n'^. Also by induction and Proposition 14.11 for ti > 2, 



det(i/„(Gi)) 



ii-l //. -.s/. ,Nx n+l-i' 



(^-l)(^ + l)-"+^-^ 



det(ffi(G„)) 



= 2-n 

1=2 

which simphfies to det(i7„(Gi)) = rt + 1. 

Example 2: Another proof of Proposition II. 2L For large Schroder paths arbitrary t, we have 

1 

^^^^ ^ l-tx-^ -x'^Fix)' 

Applying T gives 

det(i7„_i(i^i)) = det(ff„(i^)), where F^{x) ^ ^ ^ 



l + tx"^ - x^Fi (x) ' 
Applying T again, we obtain 

(1 + t)"det(i/„_i(F2)) = det(ff„(Fi)), where ^2(2;) ^ 



l-fa;2-a;2F2(x)' 

This implies F2 — F, and hence the recurrence det(i?„(F)) = (l + t)"-^ det{Hn^2{F)), with initial condition 
det(iJi(F)) = 1, and det(iJ2(F)) = l+t. 

Example 3: Another proof of Proposition 12.11 Consider the continued fraction 

1 



Fix) = 



1 — tx — xF{x) ' 

where F{x) is the generating function for the Catalan numbers for t — and the large Schroder numbers for 
t = 1. 

Under Proposition l4.2f i^ we have a unique Gi such that Gi(x) = F{x) and det(iJ^_ 2(^*1)) — det(_ff„(i^)). 
Taking G2 = (Gi(a;) -l)/x= {F{x) - l)/x, we have 

jl+t) 

' 1 - {2 + t)x - x^G2{x) 

where det(iJ„_i(G2)) = det(i7i_i(F)) and ii(a;) (1 - {2 + t)x)/{l + t). 
Under Proposition 14. II we have a unique G3 

^^^"^^ " l-{2 + t)x-{l + t)x^G3{x)' 

with G3(:e) - G2/(l + t) and det(i/„_i(G3)) = (1 + t)"'""'^ det(iI„_i(G2)). 

Under Proposition 14. 2r ii') we have a unique G4 such that G4{x) = (1 + t)G3{x) and det(i?„_2(G4)) = 
det(ff„_i(G3)). 

We see that Gi{x) = G2(x); thus det(i7„_i(G2)) = (1 + i)"-Met(iJ„_2(G2)) with det(iJi(G2)) = l + t. 
Hence det(ff„(^^)) = det(i7„_i(G2)) = (1 + 

Example 4: Another proof of Proposition 12.31 To compute det(_ff;^(F)), first we consider 
H^XF) = H„{Fi), where Fi - + ^^"^ 



1 - (2 + i)a;2 - x^Fi ' 
Applying T shows that det(i7„(Fi)) = -(1 + t)" det(i7„_2(-Fi))- 
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Example 5: For £ = 3, recall the functional equation 

^"^"'^ " 1 - te3 - x'^Foix) ■ 

For arbitrary i, our transformation gives more and more complicated expressions. This is not surprising 
since the Hankel determinants do not factor nicely. However, for < = 1 and for k — 1,2, 3, the transformation 
gives nice results similar to that of Proposition indeed, sequences of det{H^jFo)) also have period 14. 
For fc = 4 there is an interesting result. 

Subexample 5i: The sequence for det{Hl{Fo)) starts with 0, -1, 0, 1, 1, 0, -1, 0, 1, 0, -1, -1, 0, 1. If we 
define Fi so that Fq{x) = 1 + xFi{x), then 

det(iJ„(Fi)) = det{H^,{Fo)), with F^ = , 3f and d = 1. 

Then applying T repeatedly so T{Fi) = Fj+i, we obtain 



det(77„_2(i^2)) - - det(H„(Fi)), where F^ = , , ^ and d=l; 

[x + — x — X'^ — x-''b2) 

det(7I„_2(F3)) = - det{Hn{F2)), where F^ = and d = 0; 

X 

det(iJ„_i(F4)) = det(i?„(F3)), where F3 = -— — 5—- and d = I; 

[1 + X — a;^j(l — X — a; -Ts) 

det(i/„_2(F5)) = -det(ff„(^^4)), where = ^ ^ f ^ 3i 



The periodicity is established by noticing that F5 = Fi and det(iJ„_7(F5)) = — det(i/„(Fi)). 

Subexample 5ii: The sequence for dei{H^{FQ)) starts with 1, 1, 1, 1, 0, 0, —1, —1, —1, —1, —1, 0, 0, 1,. If 
we define Go so that Fq{x) = 1 + x^Gq{x), then det(i7„(G'o)) = Aet{Hl{Fo)), 

l + x 

° 1 - 2x2 - x3 - ^iQ^ ■ 

One can establish the periodicity using Proposition l4.2l However, this generating function has appeared in 
Transformation 3 of section |31 where one can see that 

(4.2) det(i?„(Go)) ^ -det{Hr,+5iFo)). 



Subexample 5iii: The sequence for det(i?^(Fo)) starts with 1, —1, —1, 0, 0, 0, —1, —1, 1, 1, 0, 0, 0, 1 and 
continues with period 14. The verification for this case uses Proposition l4.2r ii') occasionally interspersed with 
Proposition Here we will only sketch the verification. By defining Fi so that Fo{x) = 1 + a;2 + x'^Fi{x), 
one finds that 

^ _ l + 2x + x'^ +x^ 
^ ^ 1 - 2x2 - a;3 - 2a;4 _ ^sp^ ■ 
For the first transformation, with F2 — TFi, we find 

^ _ 1 - 2x + x^ 

^ ^ -1 + 4.x2 + x3 + 2x4 - x2(l + 2x + x2 + x3)i^2 ' 

in which u{x) = (— 1 + 4x2 + x'^ + 2x'')/(l — 2x + x^). Now, since u{0) = —1, one needs to apply Proposition 
14.11 for the next transformation. One proceeds until a generating function equal to Fi appears to establish 
the periodicity. We remark that d — for each transformation until the final one which uses Proposition 
I4.2f ii') with d ~ 3 (This corresponds to a fourth order block) . 

Subexample 5iv: The sequence for det(i?^(Fo)) begins with 



2, 3, 4, 0, 0, -4, -5, -6, -7, -8, 0, 0, 8, 9, 10, 11, 12, 0, 0, -12, -13, -14, -15, -16, 0, 0, 16, 
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For n > 8, an essence of periodicity can be gleaned from the recurrence 

det(i?,t(^^o)) - 4dct(i/„_i(Fo)) - dct{H^_,{Fo)), 

for which we sketch a proof, often omitting the functional equations. 

We will be applying the transformation T eight times, alternating its definition to be first under 



Proposition 14.11 and then under Proposition I4.2r ii'). Let Fi satisfy Fq 
det(77„(Fi)) = det(iJ4(Fo)), and 



1 



2 + 3.T + 2a;2 + 2x^ 



X 



Here u(0) 
d = 0, where d is for F2. With F^ 



1 - 2x2 - j;3 - _ 2.t5 - x^Fi ' 
i, where u{x) is for Fi. Thus, with F2 ^ TFi, det(i7„(F2)) 



X Fi. Hence, 



(i)"det(ff„(Fi)). Now 



TF2, det(iJ„_i(^^3)) = det(H„(^^2)). 



Here m(0) = |, where u{x) is for F3. Thus, with F4 = TFg, det(i7„_i(F4)) 



(|r-idet(ff„_i(i^3)). 



Now d = 0, where d is for F4. With F5 = TF^, det(i/„_2(i^5)) = det(£r„_i(i^4)) 

Here m(0) = |, where u(a;) is for F5. Thus, with Fq = TF5, det(ir„_2(F6)) = (|) 
Now d = 0, where d is for Fq. With F7 = TFq, detiH^siFr)) = det(ff„_2(^6)) 



3 

9^n-2 



det(i7„_2(i^5)). 



Here w(0) = |, where is for F7. Thus, with Fg = TFj, det(i7„_3(F8)) = det(i?„„3(^7))- 



Now d = 2, where d is for Fg. With Fg = TFg, det(iJ„_6(-F9)) 

/5 6 7 8 ri r, _8 _9 _10 \ 
\ 4.T 4^ 4.1 4.7 ^1 4: 41 4 T • ■ •)• 

Thus, (surprisingly) 



-dct((i/„^3(J^8)) 



(4.3) 

Moreover, 



Fo = 



det(iJ„_6(i^9)) 



20 + 16x - _ 4^3 



-(^)"(^)"-'(^)"-'(^r-'det(ij;^(Fo)) 



1 



det(ij4(^^o)) 







2x'^ + 3x-^ + 6x'^ + lOx" 



8(2 - 4x2 _ ^ 2;4) - IGx^Fg 

It is easily verified that -f9(x) and ^ + Go{x), where Go appears in Subexample 5ii, satisfy the same functional 
equation, and hence are equal. Therefore, 

det(i/„_6(i^9)) = 



2^^9(2;) - yFgiy) 



y 



n — 6 



xGo(x) - yGoiy) 



n— 6 



1 



det{H^_7{Fo)) + det(i/„_6(Go)) 



where j det(7J^_7(Fo)) is j times the determinant of the 1,1-minor of Hn-e{Go), equivalently of H^_q{Fo 



Combining this with identity H4.3|l and noting det(7J„(G'o)) 
recurrence of this subexample. 



- det(iJ„4-5(Fo)) from 1)4.2(1 proves the initial 
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